Deployment

Prepare the deployment

To run terraform by referencing our modules from your own you'll need to start by creating a terraform repo, or utilizing one you already have. We recommend using the terraform 1.0 commons module module. This will provision all the necessary config for a production gen3 instance. If you just want to create a more basic development cluster you can use this module instead.

Once you decide the module you want to use you'll want to create a .tf file that references our module, making sure to include the reference to the github repo as well as the variables you'll be passing into the module. Since you will likely want to use the module as is, you will likely want to pass in every variable. It should look similar to the following.

provider "aws" {}

terraform {
  backend "s3" {
    bucket = "s3 bucket"
    encrypt = "true"
    key = "directory in bucket/terraform.tfstate"
    region = "us-east-1"
  }
}


module "commons" {
  source = "git::github.com/uc-cdis/gen3-terraform.git//tf_files-1.0/aws/commons?ref=master"

  vpc_name=var.vpc_name
  iam_role_name=var.iam_role_name
  vpc_cidr_block=var.vpc_cidr_block
  vpc_flow_logs=var.vpc_flow_logs
  vpc_flow_traffic=var.vpc_flow_traffic
  aws_region=var.aws_region
  aws_cert_name=var.aws_cert_name
  csoc_account_id=var.csoc_account_id
  peering_cidr=var.peering_cidr
  fence_db_size=var.fence_db_size
  sheepdog_db_size=var.sheepdog_db_size
  indexd_db_size=var.indexd_db_size
  db_password_fence=var.db_password_fence
  db_password_peregrine=var.db_password_peregrine
  db_password_sheepdog=var.db_password_sheepdog
  db_password_indexd=var.db_password_indexd
  portal_app=var.portal_app
  fence_snapshot=var.fence_snapshot
  peregrine_snapshot=var.peregrine_snapshot
  sheepdog_snapshot=var.sheepdog_snapshot
  indexd_snapshot=var.indexd_snapshot
  fence_db_instance=var.fence_db_instance
  sheepdog_db_instance=var.sheepdog_db_instance
  indexd_db_instance=var.indexd_db_instance
  hostname=var.hostname
  kube_ssh_key=var.kube_ssh_key
  kube_additional_keys=var.kube_additional_keys
  hmac_encryption_key=var.hmac_encryption_key
  sheepdog_secret_key=var.sheepdog_secret_key
  sheepdog_indexd_password=var.sheepdog_indexd_password 
  sheepdog_oauth2_client_id=var.sheepdog_oauth2_client_id
  config_folder=var.config_folder
  sheepdog_oauth2_client_secret=var.sheepdog_oauth2_client_secret
  ami_account_id=var.ami_account_id
  squid_image_search_criteria=var.squid_image_search_criteria
  peering_vpc_id=var.peering_vpc_id
  squid-nlb-endpointservice-name=var.squid-nlb-endpointservice-name
  slack_webhook=var.slack_webhook
  secondary_slack_webhook=var.secondary_slack_webhook
  alarm_threshold=var.alarm_threshold
  csoc_managed=var.csoc_managed
  csoc_peering=var.csoc_peering
  mailgun_api_key=var.mailgun_api_key 
  mailgun_smtp_host=var.mailgun_smtp_host
  mailgun_api_url=var.mailgun_api_url
  fence_ha=var.fence_ha
  sheepdog_ha=var.sheepdog_ha
  indexd_ha=var.indexd_ha
  fence_maintenance_window=var.fence_maintenance_window
  sheepdog_maintenance_window=var.sheepdog_maintenance_window
  indexd_maintenance_window=var.indexd_maintenance_window
  fence_backup_retention_period=var.fence_backup_retention_period
  sheepdog_backup_retention_period=var.sheepdog_backup_retention_period
  indexd_backup_retention_period=var.indexd_backup_retention_period
  fence_backup_window=var.fence_backup_window
  sheepdog_backup_window=var.sheepdog_backup_window
  indexd_backup_window=var.indexd_backup_window
  engine_version=var.engine_version
  fence_auto_minor_version_upgrade=var.fence_auto_minor_version_upgrade
  indexd_auto_minor_version_upgrade=var.indexd_auto_minor_version_upgrade
  sheepdog_auto_minor_version_upgrade=var.sheepdog_auto_minor_version_upgrade
  users_bucket_name=var.users_bucket_name
  fence_database_name=var.fence_database_name
  sheepdog_database_name=var.sheepdog_database_name
  indexd_database_name=var.indexd_database_name
  fence_db_username=var.fence_db_username
  sheepdog_db_username=var.sheepdog_db_username
  indexd_db_username=var.indexd_db_username
  fence_allow_major_version_upgrade=var.fence_allow_major_version_upgrade
  sheepdog_allow_major_version_upgrade=var.sheepdog_allow_major_version_upgrade
  indexd_allow_major_version_upgrade=var.indexd_allow_major_version_upgrade
  ha-squid_instance_type=var.ha-squid_instance_type
  ha-squid_instance_drive_size=var.ha-squid_instance_drive_size
  deploy_single_proxy=var.deploy_single_proxy
  ha-squid_bootstrap_script=var.ha-squid_bootstrap_script
  ha-squid_extra_vars=var.ha-squid_extra_vars
  branch=var.branch
  fence-bot_bucket_access_arns=var.fence-bot_bucket_access_arns
  deploy_ha_squid=var.deploy_ha_squid
  ha-squid_cluster_desired_capasity=var.ha-squid_cluster_desired_capasity
  ha-squid_cluster_min_size=var.ha-squid_cluster_min_size
  ha-squid_cluster_max_size=var.ha-squid_cluster_max_size
  deploy_sheepdog_db=var.deploy_sheepdog_db
  deploy_fence_db=var.deploy_fence_db
  deploy_indexd_db=var.deploy_indexd_db
  single_squid_instance_type=var.single_squid_instance_type
  network_expansion=var.network_expansion
  rds_instance_storage_encrypted=var.rds_instance_storage_encrypted
  fence_max_allocated_storage=var.fence_max_allocated_storage
  sheepdog_max_allocated_storage=var.sheepdog_max_allocated_storage
  indexd_max_allocated_storage=var.indexd_max_allocated_storage
  activation_id=var.activation_id
  customer_id=var.customer_id
  fips=var.fips
  ignore_fence_changes=var.ignore_fence_changes
  ignore_sheepdog_changes=var.ignore_sheepdog_changes
  ignore_indexd_changes=var.ignore_indexd_changes
  prevent_fence_destroy=var.prevent_fence_destroy
  prevent_sheepdog_destroy=var.prevent_sheepdog_destroy
  prevent_indexd_destroy=var.prevent_indexd_destroy
  deploy_alarms=var.deploy_alarms
  ec2_keyname=var.ec2_keyname
  instance_type=var.instance_type
  jupyter_instance_type=var.jupyter_instance_type
  workflow_instance_type=var.workflow_instance_type
  secondary_cidr_block=var.secondary_cidr_block
  users_policy=var.users_policy
  worker_drive_size=var.worker_drive_size
  eks_version=var.eks_version
  workers_subnet_size=var.workers_subnet_size
  bootstrap_script=var.bootstrap_script
  jupyter_bootstrap_script=var.jupyter_bootstrap_script
  kernel=var.kernel
  jupyter_worker_drive_size=var.jupyter_worker_drive_size
  workflow_bootstrap_script=var.workflow_bootstrap_script
  workflow_worker_drive_size=var.workflow_worker_drive_size
  cidrs_to_route_to_gw=var.cidrs_to_route_to_gw
  organization_name=var.organization_name
  jupyter_asg_desired_capacity=var.jupyter_asg_desired_capacity
  jupyter_asg_max_size=var.jupyter_asg_max_size
  jupyter_asg_min_size=var.jupyter_asg_min_size
  workflow_asg_desired_capacity=var.workflow_asg_desired_capacity
  workflow_asg_max_size=var.workflow_asg_max_size
  workflow_asg_min_size=var.workflow_asg_min_size
  iam-serviceaccount=var.iam-serviceaccount
  domain_test=var.domain_test
  deploy_workflow=var.deploy_workflow
  secondary_availability_zones=var.secondary_availability_zones
  deploy_jupyter=var.deploy_jupyter
  dual_proxy=var.dual_proxy
  single_az_for_jupyter=var.single_az_for_jupyter
  oidc_eks_thumbprint=var.oidc_eks_thumbprint
  sns_topic_arn=var.sns_topic_arn
  fips_ami_kms=var.fips_ami_kms
  fips_enabled_ami=var.fips_enabled_ami
  availability_zones=var.availability_zones
  deploy_eks=var.deploy_eks
  deploy_es=var.deploy_es
  ebs_volume_size_gb=var.ebs_volume_size_gb
  encryption=var.encryption
  es_instance_type=var.es_instance_type
  es_instance_count=var.es_instance_count
  es_version=var.es_version
  es_linked_role=var.es_linked_role
  cluster_identifier=var.cluster_identifier
  cluster_instance_identifier=var.cluster_instance_identifier
  cluster_instance_class=var.cluster_instance_class
  cluster_engine=var.cluster_engine
  cluster_engine_version=var.cluster_engine_version
  master_username=var.master_username
  storage_encrypted=var.storage_encrypted
  apply_immediate=var.apply_immediate
  engine_mode=var.engine_mode
  serverlessv2_scaling_min_capacity=var.serverlessv2_scaling_min_capacity
  serverlessv2_scaling_max_capacity=var.serverlessv2_scaling_max_capacity
  skip_final_snapshot=var.skip_final_snapshot
  final_snapshot_identifier=var.final_snapshot_identifier
  backup_retention_period=var.backup_retention_period
  preferred_backup_window=var.preferred_backup_window
  password_length=var.password_length
  deploy_aurora=var.deploy_aurora
  deploy_rds=var.deploy_rds
  use_asg=var.use_asg
  use_karpenter=var.use_karpenter
  karpenter_version=var.karpenter_version
  deploy_cloud_trail=var.deploy_cloud_trail
  send_logs_to_csoc=var.send_logs_to_csoc
  route_table_name=var.route_table_name
  eks_public_access=var.eks_public_access
  deploy_gen3=var.deploy_gen3
  ambassador_enabled=var.ambassador_enabled
  arborist_enabled=var.arborist_enabled
  argo_enabled=var.argo_enabled
  audit_enabled=var.audit_enabled
  aws-es-proxy_enabled=var.aws-es-proxy_enabled
  dbgap_enabled=var.dbgap_enabled
  dd_enabled=var.dd_enabled
  dictionary_url=var.dictionary_url
  dispatcher_job_number=var.dispatcher_job_number
  fence_enabled=var.fence_enabled
  guppy_enabled=var.guppy_enabled
  hatchery_enabled=var.hatchery_enabled
  indexd_enabled=var.indexd_enabled
  indexd_prefix=var.indexd_prefix
  ingress_enabled=var.ingress_enabled
  manifestservice_enabled=var.manifestservice_enabled
  metadata_enabled=var.metadata_enabled
  netpolicy_enabled=var.netpolicy_enabled
  peregrine_enabled=var.peregrine_enabled
  pidgin_enabled=var.pidgin_enabled
  portal_enabled=var.portal_enabled
  public_datasets=var.public_datasets
  requestor_enabled=var.requestor_enabled
  revproxy_arn=var.revproxy_arn
  revproxy_enabled=var.revproxy_enabled
  sheepdog_enabled=var.sheepdog_enabled
  slack_send_dbgap=var.slack_send_dbgap
  ssjdispatcher_enabled=var.ssjdispatcher_enabled
  tier_access_level=var.tier_access_level
  tier_access_limit=var.tier_access_limit
  usersync_enabled=var.usersync_enabled
  usersync_schedule=var.usersync_schedule
  useryaml_s3_path=var.useryaml_s3_path
  wts_enabled=var.wts_enabled
  fence_config_path=var.fence_config_path
  useryaml_path=var.useryaml_path
  gitops_path=var.gitops_path
  google_client_id=var.google_client_id
  google_client_secret=var.google_client_secret 
  fence_access_key=var.fence_access_key
  fence_secret_key=var.fence_secret_key
  upload_bucket=var.upload_bucket
  namespace=var.namespace
  secrets_manager_enabled=var.secrets_manager_enabled
  ci_run=var.ci_run
}

Next you'll need to define the variables you just passed into our module, since this is technically a new terraform module. So you'll need to make a variables.tf file that should be an exact clone of the variables.tf file of our variables.tf file. If you are adding this to an existing module then you can just append our variables to your current file.

Once that is all setup you can proceed to running terraform.

Running terraform

Terraform will run in 2 parts, an initial plan and then an apply. This allows you to see the changes to your infrastructure to prevent unwanted changes that may affect the uptime of your application. Initially it should show only additions, but as you perform maintenance you will likely want to look at the plan more closely to ensure important resources are not deleted or modified.

To create a terraform plan, you can run the following.

terraform plan -var-file config.tfvars

Terraform should automatically pull in all the configuration from any files with a .tf extension in the current directory, as well as all references to the modules referenced in those files. The -var-file flag says to use the variables within the config.tfvars as variables within the module. After running this you should see a plan with a few hundred resources being added to your account.

The next step is to apply these changes if you're happy with them. To do that you can run a similar command, but change plan to apply.

terraform apply -var-file config.tfvars

It will ask you if you want to commit these changes and if you approve them then terraform will run for a while and create the resources.

Note

Terraform may time out creating resources or run into errors. If it does, it can be normal due to long provisioning processes and race conditions. If you see errors run apply a couple more times and if the errors persist, reach out over our community slack channel ffor assistance.